Handle pushes containing multiple backouts

[marco-c]

If a failure is marked as fixed by commit of a backout push which contains multiple backouts, we can't really know which of the backouts is actually relevant.

From Matrix:

marco
Aryx: I'm often seeing 'fixed by commit' information pointing to the wrong commit when a backout push contains two backouts

Aryx
yes, because TH only offers the push head

marco
ah, so that means it's consistent
I wonder if I can find an automated way to figure it out

Aryx
I copy the correct commit if I can mark all in one go, but if the trickle in, that's really cumbersome
where it started and if the failing test got touched will get you close
the backout commit messages also mention the failing test or a part of the failure message (e.g. assert)

marco
unfortunately it's not easy to know where it started, unless the failures are all backfilled (but afaics they are not always)
looking at the logs to get the failure message might be expensive, unless that's available in ActiveData (I'll ask ekyle)

[marco-c]

For example, a0d10a7e5c3f219e8346667b107cb02506cbbdd8 backs out both 48a854ee6710ee77fce1b7dc08799352d6ff3fdb and 379e2c6c049115d64977587961d08b42a09ad3c9.

build-linux64-fuzzing/debug is due to 379e2c6c049115d64977587961d08b42a09ad3c9, while build-android-x86_64-asan-fuzzing/opt is due to 48a854ee6710ee77fce1b7dc08799352d6ff3fdb, but they:

• are both classified as fixed by commit a0d10a7e5c3f219e8346667b107cb02506cbbdd8;
• both only run and failed in 48a854ee6710ee77fce1b7dc08799352d6ff3fdb.

The backout of 48a854ee6710ee77fce1b7dc08799352d6ff3fdb is b45a70689704ffc67766812444530cb15005eb45 (which is part of the a0d10a7e5c3f219e8346667b107cb02506cbbdd8 push).

[marco-c]

Some additional examples that we currently don't handle perfectly:

• ccd3a06efe81123e9ff32a697bc3c0f54e26a734 (we find a lot of false positives because of bad classifications);
• 7daef473f5ebb2fce0b58a82972cf7b4a8079ae8
• 332e06a545d2667fd940d5bf0150bb372146559e
• 943fe621074459d34144ab7c0af07d7443ba5aed
• a845717e4d10ad7cb65ca64b08666c373db6e1de
• a360635bc81723e26ff35509ee5a5fd38268a5b6

When we get to fixing this, we can test with the pushes above.

[marco-c]

See also #198.

[marco-c]

See also https://bugzilla.mozilla.org/show_bug.cgi?id=1670708.

[marco-c]

As part of this, we should add a test where we have two backouts of push A and B, X passes on A but fails on B, fixed by commit points to the backout push (X should be attributed to B but not to A).